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Indonesian automatic text summarization research is developed rapidly. The 
quality, especially readability aspect, of text summary can be reached if the 
meaning of the text can be maintained properly. Therefore, this research aims 
to enhance the quality of extractive Indonesian automatic text summarization 
with considering the quality of structured representation of text. This research 
uses sequential pattern mining (SPM) to produce This research use SPM to 
produce sequence of words (SoW) as structured text representation using 
PrefixSpan algorithm. Then, SPM is combined with feature-based approach 
using sentence scoring method to produce summary. The experiment result 
using IndoSum dataset shows that even though the combination of SPM and 
sentence scoring can increase the precision value of recall-oriented 
understudy for gisting evaluation (ROUGE)-1, ROUGE-2, and ROUGE-L, 
from 0.68 to 0.76, 0.54 to 0.69, and 0.51 to 0.72. Especially, combination of 
SPM and Sentence Scoring can enhance precision, recall, and f-measure of 
ROUGE-L that consider the order of word occurance in measurement. SPM 
increases ROUGE-L f-measure value of sentence scoring from 0.32 to 0.36. 
Moreover, combination of sentence scoring and SPM is better than SumBasic 
that used as feature-based approach in the previous Indonesian text 
summarization research. 


This is an open access article under the CC BY-SA license. 


Corresponding Author: 
Dian Sa’adillah Maylawati 


Centre for Advanced Computing Technology, Faculty of Information and Communication Technology 


Universiti Teknikal Malaysia Melaka 


Street of Hang Tuah Jaya, Melaka, 76100, Malaysia 


Email: diansm @uinsgd.ac.id 


1. INTRODUCTION 


The development of automatic text summarizing techniques is accelerating. natural language 
processing includes automatic text summarization (NLP) [1]. The study of natural language processing, or 
NLP, is a subfield of linguistics, computer science, and artificial intelligence that focuses on how to build 
computers that can process and analyze massive amounts of natural language input [2]-[4]. In general, there 
are two forms of automatic text summarization: extractive and abstractive [5]. Extractive summarization 
creates a sequential summary based on the document's source and without changing the word structure of the 
sentences [6], [7]. The source document sentences make up the final summary. Extractive summaries or 
extracts are made by detecting key sentences in the source content and selecting them directly. While 
abstractive summarization results in a modified summary [8], for example paraphrase. As a result, the 
abstracted summary does not contain the same phrases or structure as the original document, but it nevertheless 


conveys the same message. 
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In today's world, there are numerous approaches for text summarizing. There are at least three 
techniques to automatically producing summary, including feature-based approach. The feature-based 
technique considers and calculates document components (such as words, sentences, phrases, and so on) 
throughout the automatic text summarizing process [9]. Sentence scoring [10], [11], SumBasic [12], and latent 
semantic analysis (LSA) [13]-[15] are feature-based approach algorithms that can be utilized for text 
summarization in the previous research. However, the production of a readable summary is a challenge in 
automatic text summarizing research [16]-[18]. It means that there isn't much of a disconnect between the 
summary result and the reader's comprehension. Because it ensures that the summary result is readable and 
understood, the readability of the summary result is a critical aspect in evaluating the effectiveness of automatic 
text summarization [19]-[21]. 

Therefore, to resolve the readability issue from the summary results, it can be started from preparing 
a quality text representation. Text representation that used must be able to maintain the meaning of the text 
data. The sequence of words (SoW) is one of widely used as text representation for text analytic, either for text 
mining, information retrieval, text similarity, even for text summarization (both for single and multi-document) 
which is proven to be able to maintain the meaning of the text well and even increase. However, most of them 
are used for English, while each language has its own peculiarities, including the Indonesian language. 
Moreover, there is no specific research has been found using a SoW as a text representation for Indonesian 
automatic text summarization. 

In previous research, the SoW used for the Indonesian language is not specific for text summarization 
[22]. While the SoW as text representation that maintains the meaning of the text and minimize the reader's 
understanding gap with the summary results, to reach readability of summary result. There is a text 
summarization study that uses the SoW for the Malaysian language [23], [24], which is one family with the 
Indonesian language, but there are still many differences. Therefore, this study aims to investigate and enhance 
the quality of Indonesian text summary result with SoW as text representation to maintain the meaning of the 
text. This research use PrefixSpan algorithm as one of sequential pattern mining (SPM) technique [25] to 
produce SoW and Sentence Scoring as feature-based approach to produce Indonesian automatic text summary. 
Then, the summary result will be evaluated with recall-oriented understudy for gisting evaluation (ROUGE- 
1), ROUGE-2, and ROUGE-L that many used in current Indonesian automatic text summarization [26]-[32]. 
This metrics is used to evaluate the quality of Indonesian automatic text summary that produced by the 
combination of SPM and sentence scoring. 


2. MATERIALS AND METHOD 

Figure 1 presents the research overview that combine SPM and sentence scoring to produce 
Indonesian automatic text summary. Combination of SPM in sentence scoring aims to enhance the performance 
of summary result and reach the readable summary. This research has several activities, begin from data 
gathering and data preparation, producing Indonesian automatic text summary using sentence scoring, 
producing Indonesian automatic text summary using combination of SPM and sentence scoring, evaluate the 
ROUGE value of summary result from sentence scoring with and without SPM, then comparative analysis 
whether SPM can enhance the quality of Indonesian automatic text summary result. 


Indonesian News Dooument 
(IndoSum dataset) 


Indonesian Automatic Text Summanization 
Feature-based Approach 


enhance 


Structured Representation of Text 


using produce Sequence of Words 
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Sentence Scoring & PrefixSpan PrefixSpan algorithm 


Figure 1. Research overview 
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Most of the related studies about Indonesian text summarization used news articles for an experiment 
because easily collected freely and online. Besides that, the most important thing is that news articles have a 
pretty good language structure or writing structure, compared to social media, which contains many languages 
with poor structure and even slang. This research uses IndoSum dataset that contain news articles from CNN 
Indonesia, Kumparan, and Merdeka.com [26]. This dataset is widely used as a benchmark for Indonesian 
automatic text summarization research. And also, the dataset have manual summary which created by two 
Indonesian native speakers as an expert. The IndoSum dataset is open access and available at Github [33]. The 
news articles dataset are classified by 6 categories: entertainment, inspiration, sport, showbiz, headline, and 
technology. Actually, from that sources total news article are 18,774 news documents. Indonesian news 
document collections are presented with JavaScript object notation (JSON) type. 

The next process is data preparation or data pre-processing that significant steps in most computational 
linguistics investigations [34]. Raw texts should be pre-processed to ensure that they have a suitable 
representation and can be used effectively in experiments. There are several text preprocessing procedures in 
this study: separating the sentences, case-folding, tokenizing, removing regular expression (non-letter 
characters), removing Indonesian topwords, and stemming process using Nazief-Adriani algorithm for 
Indonesian language. In addition, there is Sastrawi library on Python programming that can be used to prepare 
Indonesian text data in the text pre-processing phase [35]. Sastrawi provides the Indonesian stop words list and 
Nazief-Adriani as a popular stemming process for Indonesian text [35], [36]. However, sometimes text pre- 
processing activity is not the same for all cases, depending on the need in each case. Sentence separation aims 
to distinguish one sentence from another according to the needs of the feature approach using sentence scoring. 

Then, there are two experiment scenarios: i) produce Indonesian text summary with feature-based 
approach using sentence scoring algorithm and bag-of-words (BoW) as structured representation of text; and 
ii) produce Indonesian text summary with combinantion of feature-based approach using sentence scoring and 
PrefixSpan algorithm as a part of SPM, and SoW as structured representation of text. 

The performance of SPM combination with feature-based is evaluated using co-selection-based 
analysis with ROUGE-1, ROUGE-2, and ROUGE-L. ROUGE also evaluate the precision, recall, and F-1 
score. ROUGE-1 and ROUGE-? is a part of ROUGE-N, where N=1, 2, 3, and 4. ROUGE-N score evaluation 
use n-gram overlaps between the candidate document and the reference documents. ROUGE-N formula is 
available in (1) [37]. 


ROUGE N= Lse(Reference Summaries} Lgramnes Countmatch(gramn) (1) 


Lse(Reference Summaries} Xgramnes Count(gramy) 


Where nv is the length of an n-gram, gramy is the maximum number of n-grams found in a candidate summary 
and a set of reference summaries, and countmatch(gramn) is the maximum number of n-grams found in a 
candidate summary and a set of reference summaries. 

ROUGE-L is a statistics program that uses the longest common subsequence, but (LCS). By 
considering similarities in sentence level structure, the longest common subsequence problem automatically 
determines the longest co-occurring in sequence n-grams. The benefit of LCS is that it prefers in-sequence 
matches that reflect sentence level word order rather than consecutive matches. Since it automatically includes 
the longest common n-grams in sequence, it is not dependent on a specified n-gram length. The ROUGE-N 
formula can be found in (2). 


LCS(gramn) 


ROUGE —L= (2) 


Count(gramn) 


Where LCS(gram,) is longest common subsequence between reference and model output. Then, gram, is the 
maximum number of n-grams found in a candidate summary and a set of reference summaries. 


3. RESULTS AND DISCUSSION 

This section presents the result of the improvement of Indonesian text summary by enhancing the 
SoW as text representation. It is critical to properly prepare text representation to achieve the readable summary 
result. This section sequentially presents the process of combination of SPM and sentence scoring, the result 
of sentence scoring in producing Indonesian text summary, the result of combination of SPM and sentence 
scoring, and discussion section. 


3.1. Combination of sequential pattern mining and sentence scoring process 
This section explains the overview of proposed method that combine sentence scoring and SPM in 
producing Indonesian text summary. Figure 2 illustrate the flow process of feature-based approach using 
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sentence scoring in producing Indonesian text summary. Figure 2(a) shows the sentence scoring process using 
word frequency, while Figure 2(b) illustrate the combination of SPM and sentence scoring in which SPM 
replace the word frequency. Sentence Scoring is combined with SPM to produce SoW as text representation. 
This research uses PrefixSpan algorithm as SPM method that has been used for Indonesian text [22], [38]. The 
sequence of word that produced from PrefixSpan will replace the word frequency in sentence scoring. 


IndoSum d IndoSum . * 
Dataset Word Frequency Dataset Sequential Pattern Mining 
Sentence Scoring 


Sentence Scoring 
Summary Result 


Summary Result 


End End 


(a) (b) 


Figure 2. Illustrate the flow process of feature-based approach using sentence scoring (a) sentence scoring 
without SPM and (b) sentence scoring with SPM 


This research use PrefixSpan algorithm as SPM technique to produce SoW. PrefixSpan or Prefix- 
projected sequential pattern mining is an SPM algorithm that adapts divide and conquer principles and pattern 
building to generate efficient sequence patterns from large sequence databases [39]. Here is the PrefixSpan 
Algorithm 1. 


Algorithm 1: PrefixSpan 

Input: A sequence database S, and the minimum support (threshold min_support) 

Output: The complete set of sequential patterns 

Method: Call PrefixSpan({},0,S) 

Subroutine: PrefixSpan(a,1,S|«) 

The parameters are 1) a is a sequential pattern; 2) 1 is the length of a; and 3) Sla is the 
a-projected database if a # {}, otherwise, it is the sequence database S. 

Method: 
1. Scan Sla once, find each frequent item, b, such that 

(a) b can be assembled to the last element of a to form a sequential pattern; or 

(b) {b} can be appended to a to form a sequential pattern. 

2. For each frequent item b, append it to a to form a sequential pattern a’, and output a’. 
3. For each a’, construct a’-projected database S|la, and call PrefixSpan(a,1+1,Sla’) 


Based on PrefixSpan algorithm that explained above, PrefixSpan will find length-1 sequential pattern 
first, then dividing the search space. Then, finding subsets of sequential patterns with prefix, respectively. The 
collection of patterns discovered during the recursive mining process is known as the set of sequential patterns. 
The illustration of combination between SPM and sentence scoring and Bellman-Ford algorithm will explain 
in detail in this chapter with the following news text example: 

In Indonesian: Jakarta, CNN Indonesia -- Gubernur Jawa Barat Ridwan Kamil meminta kepolisian 
untuk memperketat pintu masuk di wilayah perbatasan, termasuk jalan-jalan tikus. Menurut Ridwan Kamil, 
Pembatasan Sosial Berskala Besar (PSBB) di wilayah Bandung raya yang sudah diberlakukan sejak Rabu 
(22/4), masih ditemui sejumlah pelanggaran. Salah satu titik yang harus diperbaiki adalah wilayah 
perbatasan."Mulai sekarang kita perketat penjagaan di perbatasan, tidak boleh ada warga yang masuk 
maupun keluar dari wilayahnya, kecuali dengan alasan yang jelas," kata pria yang karib disapa Emil itu saat 
menggelar pertemuan video conference dari Gedung Pakuan, Kota Bandung, Sabtu (25/4). 

In English: Jakarta, CNN Indonesia - West Java Governor Ridwan Kamil has asked the police to 
tighten entrances in the border area, including rat roads. According to Ridwan Kamil, Large-scale Social 
Restrictions (PSBB) in the Bandung area, which had been imposed since Wednesday (22/4), several violations 
were still encountered. One of the points that need to be improved is the border area. "From now on, we tighten 
security at the border, no residents may enter or leave the area, except for obvious reasons," said the close 
friend Emil when holding a video conference meeting from Pakuan Building, Bandung City, Saturday (25/4). 
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Before news text is processed in sequential pattern layer, the news text is prepared and cleaned at the 
pre-processing stage, including case folding, removing regular expression, removing stop words, stemming, 
and also separating sentence for the needs of the sentence classification to be selected in the result of summary. 
From the news text example above, the following results are obtained for text pre-processing and sequential 
patterns (form minimum of 25% of minimum support or threshold value) as shown in: 
<jakarta cnn indonesia> 
<gubernur jawa barat ridwan kamil pinta polisi ketat pintu masuk wilayah batas masuk jalan jalan tikus> 
<turut ridwan kamil batas sosial skala besar psbb wilayah bandung raya sudah laku sejak rabu masih temu 
jumlah langgar> 
<salah satu titik harus baik wilayah batas> 
<mulai ketat jaga batas boleh ada warga masuk mau keluar wilayah alas jelas kata pria karib sapa emil gelar 
temu video conference gedung pakuan kota bandung sabtu> 
The example of Sequential Pattern that produced: 
<{ketat}>: 2 
<{temu}>: 2 
<{masuk}>: 3 
<{wilayah}>: 4 
<{batas}>: 4 
<{ketat, masuk}>: 2 
<{ketat, wilayah}>: 2 
<{masuk, wilayah}>: 2 
<{batas, wilayah}>: 2 
<{wilayah, batas}>: 2 
<{ketat, masuk, wilayah}>: 2 
Etc. 


3.2. Combination of sequential pattern mining and sentence scoring process 

There are many features that used in sentence scoring method. These characteristics have been used 
in a variety of strategies, either alone or in conjunction with other techniques. There are at least 22 features 
used in sentence scoring, including [40] terms like: word similarity among sentences, cue word, title similarity, 
proper noun, word co-occurance, font style, lexical similarity, term frequency, sentence location, TF-IDF, 
sentence similarity, numerical value, TextRank, sentence length, positive keyword, negative keyword, busy 
path, aggregate similarity, word similarity among paragraphs, iterative query. This research use term frequency 
(TF) or word for sentence scoring method. Term frequency is the first time it was proposed for extracting 
sentences from text documents. Sentences were graded based on the frequency of terms in the sentence. Term 
frequency has a greater impact on final scoring. 

Term frequency, term in this research is word. So, word frequency is used to extract each word from 
document and the frequency of occurrence in the document is calculated. This study makes use of 
normalization TF, which compares a term's frequency to the highest value for the entire or a group of terms in 
a document. As shown in (3) shows calculation of term frequency. 
Wordgoc 


TF = (3) 


Where Wordac means the word frequency that appears in the document, while Max_Word4doc is maximum 

frequency word that appears in the document. 

Sentence scoring method in this research is a feature-based method (basic method for text 
summarization) that use word frequency. Where, each word receives a score and the weight of each sentence 
is the sum of all scores of its constituent words [41], [42]. So, the algorithm begins from: 

- Calculate a word frequency to see how often it appears in the text. In other words, the final summary is 
more likely to include sentences that contain the most common words in the document. The idea is that a 
word's likelihood of indicating the text's subject increases with its frequency in the text. 

-  Tokenize the sentences into words by looping through each sentence in the collection of sentences. 

- Ifthe sentence is missing from the sentence list, the weighted frequency of the first word in the sentence 
should be used as the sentence's value. On the other hand, if the sentence already exists, then increase the 
value by the word's weighted frequency. 

7 Sentence score in (4). 


Sente€NC€ score = >, word_frequency_in_sentence (4) 


The example of sentence scoring method with the example document is available: 
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S1: <jakarta, cnn, indonesia> 
S2: <gubernur, jawa, barat, ridwan, kamil, pinta, polisi, ketat, pintu, masuk, wilayah, batas, masuk, jalan, 
jalan, tikus> 
$3: <turut, ridwan, kamil, batas, sosial, skala, besar, psbb, wilayah, bandung, raya, sudah, laku, sejak, 
rabu, masih, temu, jumlah, langgar> 
S4: <salah, satu, titik, harus, baik, wilayah, batas> 
S5: <mulai, ketat, jaga, batas, boleh, ada, warga, masuk, mau, keluar, wilayah, alas, jelas, kata, pria, karib, 
sapa, emil, gelar, temu, video, conference, gedung, pakuan, kota, bandung, sabtu> 


- Term/word frequency: 


jakarta: 1 -> %4 = 0.25 
enn: | -> '4=0.25 


indonesia: 1 -> %4=0.25 
gubernur: | -> %4=0.25 


jawa: 1 -> %4=0.25 
barat: 1 -> %4=0.25 


ridwan: 2 -> 2/4=0.5 


kamil: 2 -> 2/4 = 0.5 
pinta: 1 -> %4=0.25 
polisi: 1 -> %4=0.25 
ketat: 1 -> %4=0.25 
pintu: 1 -> 4=0.25 
masuk: 2 -> 2/4 = 0.5 
wilayah: 4 -> 4/4=1 
batas: 4 -> 4/4=1 


jalan: 2 -> 2/4=0.5 
tikus: 1 -> %4=0.25 
turut: 1 -> 4=0.25 
sosial: 1 -> 4=0.25 
skala: 1 -> %4=0.25 
besar: 1 -> 4=0.25 
psbb: 1 -> %4=0.25 


bandung: 2 -> 2/4=0.5 


raya: 1 -> '%4=0.25 
sudah: 1 -> 4=0.25 
laku: 1 -> 4 = 0.25 
sejak: 1 -> %4 = 0.25 
rabu: 1-> 4 =0.25 
masih: | -> %4=0.25 
temu: 2 -> 2/4=0.5 


jumlah: 1 -> %4=0.25 
langgar: 1 ->'4=0.25 
salah: 1 -> '%4=0.25 
satu: 1 -> 4=0.25 
titik: 1 -> %4=0.25 
harus: 1 -> 4=0.25 
baik: 1 -> 4 = 0.25 
mulai: 1 -> %4=0.25 
ketat: 2->%=0.5 
jaga: 1 -> %4=0.25 
boleh: 1 -> 4 = 0.25 
ada: 1 ->'%4=0.25 
warga: 1 -> '4=0.25 
mau: | -> 4=0.25 
keluar: 1 -> %4=0.25 


alas: 1 -> %4=0.25 
jelas: 1 -> %4=0.25 
kata: 1 -> %4=0.25 
pria: 1 -> %4=0.25 
karib: 1 -> '4= 0.25 
sapa: 1 -> %=0.25 
emil: 1 -> %4=0.25 
gelar: 1 -> '%4=0.25 
video: 1 -> %4=0.25 


conference: | -> 4=0.25 


gedung: 1 -> '4=0.25 
pakuan: | -> 4 =0.25 
kota: 1 -> %4=0.25 
sabtu: 1 -> %4=0.25 


Sentence scoring: 

S1: 0.25+0.25+0.25 = 0.75 

$2: 0.25+0.25+0.5+0.5+0.25+0.2540.25+0.5+0.5+0.5+0.5+0.25=5.0 

$3: 0.254+0.5+0.5+1+0.25+0.25+0.25+0.25+1+0.5+0.25+0.25+0.25+0.25+0.25+0.25+0.5+0.25+0.25= 
7.25 

S4: 0.254+0.25+0.25+0.25+0.25+14+1=3.25 

$5:0.25+0.25+0.25+1+0.25+0.25+0.25+0.5+0.25+0.25+1+0.25+0.25+0.25+0.25+0.25+0.25+0.25+ 
0.25+0.25+0.25+0.25+0.25+0.25+0.25+0.5+0.25=8.75 


- Summary result: 

Sentence scoring method will produce the summary based on the maximum number of sentences that 
want to generate. For example from news text in the subsection 3.1, if maximum sentences in summary is two, 
then the result of summary is sentence three (S3) and sentence five (S5). The result is “Menurut Ridwan Kamil, 
Pembatasan Sosial Berskala Besar (PSBB) di wilayah Bandung raya yang sudah diberlakukan sejak Rabu 
(22/4), masih ditemui sejumlah pelanggaran. "Mulai sekarang kita perketat penjagaan di perbatasan, tidak 
boleh ada warga yang masuk maupun keluar dari wilayahnya, kecuali dengan alasan yang jelas," kata pria 
yang karib disapa Emil itu saat menggelar pertemuan video conference dari Gedung Pakuan, Kota Bandung, 
Sabtu (25/4).” In English: (According to Ridwan Kamil, Large-scale Social Restrictions (PSBB) in the 
Bandung area, which had been imposed since Wednesday (22/4), several violations were still encountered. 
One of the points that need to be improved is the border area. "From now on, we tighten security at the border, 
no residents may enter or leave the area, except for obvious reasons," said the close friend Emil when holding 
a video conference meeting from Pakuan Building, Bandung City, Saturday (25/4)). 


3.3. Combination of sequential pattern mining and sentence scoring process 

The experiment of proposed method which combine SPM with sentence scoring and Bellman-Ford is 
conducted using Python with 18,774 news documents that open access and available (named IndoSum)-[26]. 
The news articles dataset are classified by 6 categories: entertainment, inspiration, sport, showbiz, headline, 
and technology. However, this experiment do not need the categorization because just need to know the 
performance of proposed method. IndoSum also provide the manual summary to be compared with the 
summary that produced by system. The experiment contain the text pre-processing including separating 
sentences, tokenizing, lowering case, removing character non-letter and regular expression, removing 
Indonesian stopwords, and stemming process using Porter algorithm for Indonesian language. Stop-words 
removing and stemming process are used Sastrawi library that commonly used for NLP research with 
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Indonesian text [35]. Then, for PrefixSpan algorithm as a part of SPM method uses prefixspan library for 
Python [43]. There are two experiments for this section as depicted in Figure 2: i) produce automatic summary 
with sentence scoring; and ii) produce automatic summary with combination of sentence scoring and SPM. 
Then, all of the result of experiment is evaluated using ROUGE evaluation metrics. ROUGE evaluation in this 
experiment uses rouge-score library for Python [44]. Last, the result of experiment will be interpreted and 
analyzed in the discussion section. 

To evaluate the performance of proposed method which combine SPM with sentence scoring, the 
evaluation process measures the result of summary using ROUGE-1, ROUGE-2 and ROUGE-L. ROUGE 
evaluation shows the performance of summary result with precision, recall, and f-measure value. Table 1 and 
Figure 3 show the average value of ROUGE-1, ROUGE-2, and ROUGE-L evaluation of each method. 

According to the ROUGE-1 evaluation results, the combination of sentence scoring and SPM has the 
highest precision, the combination of Bellman-Ford and SPM has the highest recall, and the combination of 
Bellman-Ford and SPM has the highest f-measure. These findings indicate that when SPM was combined with 
Sentence Scoring, the precision value increased. The ROUGE-2 result, like the ROUGE-1 result, indicates that 
SPM can improve the precision of sentence scoring results. The ROUGE-L evaluation yields a very different 
result. The combination of sentence scoring and SPM improved all metrics, including recall and f-measure. 

Sentence scoring with IndoSum dataset yields an average f-measure of 0.44 for ROUGE-1, 0.34 for 
ROUGE-?, and 0.32 for ROUGE-L. When sentence scoring is combined with SPM, the f-measure of ROUGE- 
1 and ROUGE-2 drops to 0.40 and 0.33, respectively, while the f-measure of ROUGE-L rises to 0.36. ROUGE- 
L, which measures long common subsequence (LCS), is more appropriate for the proposed method with SPM. 
Because SPM generates sequence patterns that pay attention to the order in which words appear, as well as an 
assessment using ROUGE-L that pays attention to the order of occurrence of words (in the form of n-grams) 
in a sequence pattern, there is no need to define the n-gram length at the start because it will be calculated 
automatically in the longest sequence pattern. As a result, while SPM did not improve the f-measures of 
ROUGE-1 and ROUGE-?2, it did improve the f-measure of ROUGE-L, which is a better metric for representing 
summary quality. 

Previous research used the modified SumBasic method with the same IndoSum dataset in their 
experiment [26]. SumBasic is a feature-based approach that ranks sentences based on word frequency and 
chooses the top sentences to serve as the summary [12], [45]. Figure 4 and Table 2 provide the comparison of 
ROUGE score between SumBasic, sentence scoring, and combination of sentence scoring and SPM. 
SumBasic's f-measure from ROUGE-1 is 0.359, ROUGE-?2 is 0.202, and ROUGE-L is 0.338. When compared 
to the performance of the proposed method, the f-measure results of ROUGE-1, ROUGE-2, and ROUGE-L 
are higher than SumBasic. This means that sentence scoring combined with SPM outperforms SumBasic. 
Furthermore, when sentence scoring is combined with SPM, the f-measure of ROUGE-L increases. 


Precision Recall F-Measure 


Sentence Scoring ROUGE-1 f Sentence Scoring ROUGE-2 
O Sentence Scoring ROUGE-L Sentence Scoring and SPM ROUGE-1 


® Sentence Scoring and SPM ROUGE-2 Sentence Scoring and SPM ROUGE-L 
Figure 3. ROUGE evaluation result of sentence scoring and proposed method 


Table 1. ROUGE evaluation result of sentence scoring and proposed method 


Method Evaluation Precision Recall | F-Measure 
ROUGE-1 0.6825 0.3187 0.4345 
Sentence scoring ROUGE-2 0.5420 0.2506 0.3427 


ROUGE-L 0.5112 0.2370 0.3238 
ROUGE-1 0.7615 0.2737 0.4026 
Sentence scoring andSPM ROUGE-2 0.6946 0.2177 0.3315 
ROUGE-L _ 0.7205 0.2377 0.3575 
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—e— SumBasic —@— SentenceScoring ==atr== Sentence Scoring and SPM 


Figure 4. Comparison of ROUGE evaluation result between proposed method and SumBasic 


Table 2. Comparison of ROUGE evaluation result between proposed method and SumBasic 
ROUGE Evaluation (Percentage of F-Measure) 


oe ROUGE-1___ ROUGE-2___ ROUGE-L 
SumBasic [26] 35.96 20.19 33.77 
Sentence scoring 43.45 34.27 32.38 
Sentence scoring andSPM _ 40.26 33.15: 35.75 


4. CONCLUSION 

This research aims to enhance the quality of Indonesian automatic text summary in the aspect of 
readability. This chapter provides the study on sequential pattern mining (SPM) that combined with feature- 
based approach, such as sentence scoring. SPM replace the word-frequency calculation in sentence scoring. 
The experiment is conducted using IndoSum dataset with four scenarios: producing Indonesian text summary 
using sentence scoring and producing Indonesian text summary using combination of sentence scoring and 
SPM. Then, the performance of proposed method is evaluated with co-selection-based analysis using ROUGE- 
1, ROUGE-2, and ROUGE-L. Based on the ROUGE evaluation result, overall SPM can improve the quality 
of summary result compared without using SPM, especially in precision and f-measure value. Precision value 
of ROUGE-1, ROUGE-2, and ROUGE-L of combination between sentence scoring and SPM is better than 
sentence scoring only. Although recall and f-measure at ROUGE-1 and ROUGE-2 in the combination of 
sentence scoring and SPM did not increase, but, at ROUGE-L both precision, recall, and f-measure increased. 
Seeing these results, combining SPM in the automatic text summary process has the potential to be further 
enhanced to improve Indonesian text summary performance. For the further works, SPM can be combined with 
the other methods to enhance the quality of Indonesian automatic text summarization. Then, it is important to 
evaluate the readability of summary result, not only using co-selection-based analysis, but also using content- 
based analysis and involve an Indonesian language expert to evaluate the summary result. 
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